perf(vulkan): decode GEMV reads four elements per load, and the row lever is dead - #184

…ever is dead

vt_matmul_vec is 89-90% of 27B decode GPU time and moves ~52 GB of bf16 weights
per token, so it was the largest single line item left. Two structural
differences against llama.cpp's mul_mat_vec (pin 237ad9b96) were the candidates.
ONE is worth about 1%, the OTHER is a measured LOSS on this device, and the win
is 7x smaller than the isolated sweep first said. All three of those are the
result, and the third is the most useful.

WHAT SHIPPED. VT_MM_PACK, a specialization constant selecting the LOAD WIDTH for
a 16-bit operand: one element per load (2 B, the previous behaviour), two through
the buffer's existing 32-bit view, or four through a new 64-bit view of the same
VkBuffer. Ported in intent from llama.cpp's data_b_v4 / dequantize4_2aligned
(vulkan-shaders/dequant_funcs.glsl:54-61, which reads bf16 weights as two 32-bit
words). Same bytes, same coalescing -- a 32-lane subgroup still covers 128 or 256
CONSECUTIVE bytes -- so this CANNOT reduce DRAM traffic. It reduces LOAD
INSTRUCTIONS, four to one. Default 2, degrading a width at a time when K or an
operand byte offset is not aligned to it, and declining for an f32 operand.

WHAT DID NOT WORK, AND ITS NUMBER. VT_MM_ROWS, llama.cpp's NUM_ROWS
(mul_mat_vec_base.glsl:90; ggml-vulkan.cpp:4705-4719 sets it per vendor via
rm_stdq), computing 2 or 4 output elements per workgroup so one loaded activation
feeds several weight rows. MEASURED 0.966x at 2 rows and 0.968x at 4, in every
pass and at every load width. It ships OFF. What it was supposed to buy was never
traffic -- the weights are read exactly once per token at any row count -- and the
activation re-reads it saves were L2 hits. Halving the workgroup count halves the
independent sequential read streams the memory controller sees while each
surviving workgroup interleaves rows K*2 bytes apart, which is the likely cost.
The axis is KEPT, off by default, because llama.cpp raises it exactly on the AMD
GCN and Intel parts this backend's portability case is aimed at, so deleting it
would mean rediscovering it on the VK-I board. It is bit-identical to rows=1 at
any value, verified by a full-output bit-hash and not only by argument.

MEASURED, ISOLATED SWEEP, AND IT OVERSTATED THE WIN BY 7x.
benchmarks/vulkan_gemv_ab.cpp runs the GEMV over the seven (k, n) shapes
Qwen3.6-27B actually dispatches; 9 arms x 4 rotated passes, each arm paired
against the rows=1/pack=0 baseline measured IN THE SAME PASS (the box drifted
15.5% peak-to-peak between passes, so an unpaired ranking would have been noise):

              pack 0     pack 1     pack 2
  rows 1      1.000x     1.025x     1.086x
  rows 2      0.966x     1.017x     1.047x
  rows 4      0.968x     1.014x     1.039x

MEASURED, REAL 27B DECODE, which is the number of record. Two-length GPU-timestamp
diff, output-len 36 minus output-len 4 over 32 decode tokens, so prefill and every
one-time cost cancels. Run twice, arm order reversed the second time:

  per decoded token        blk1 pack0  blk1 pack2  blk2 pack0  blk2 pack2
  vt_matmul_vec ms           211.99      209.52      214.59      210.38
  GB/s (52.1 GB/token)        245.8       248.7       242.8       247.6
  % of the 273 GB/s roof      90.0%       91.1%       88.9%       90.7%
  speedup                                 1.012x                  1.020x

1.012x and 1.020x, not 1.086x. THE SWEEP WAS IN THE WRONG REGIME: it re-reads ONE
356 MB weight buffer 320 times, so its DRAM rows and TLB entries stay hot and
instruction issue becomes visible as a secondary constraint. Decode streams 50 GB
of distinct weights once per token, where DRAM is the whole story. The campaign
already has "a negative result is regime-dependent" on record from the GEMV
unroll; this is the same lesson running the other way, and an isolated harness
must be shown to reproduce the e2e operating point (this one does: 89.3% of roof
against the e2e diff's 90.0%) before its RANKING is believed.

END TO END. Order-alternated AB/BA, page cache dropped before every leg, two
blocks of 6 and 8 pairs, one binary with VT_VULKAN_GEMV_PACK as the only variable
(=0 reproduces the pre-row kernel bit for bit). Eight of the 28 legs landed in the
lm_head slow mode described below, 5 baseline and 3 pack=2, so arm-symmetric; they
are reported rather than dropped. Of the six pairs where BOTH legs were clean,
pack=2 wins 5, median paired ratio 1.0064x. Clean-leg medians 242.38 -> 240.54 ms,
decode 4.126 -> 4.157 tok/s against llama.cpp Vulkan's 4.35. The measured noise
floor on the clean legs is 0.88% peak-to-peak on the baseline arm (9 legs) and
1.92% on the other (11), so a 0.7% effect sits AT the edge of what an e2e wall
clock resolves here; the GPU-timestamp diff is the instrument that resolves it,
and the two agree in sign and magnitude.

SO: IS ANYTHING LEFT IN THE GEMV? ABOUT ONE PERCENT. The kernel was already at
90.0% of the GB10 bandwidth roof and is at 91.1%. llama.cpp Vulkan's 4.35 tok/s is
229.9 ms/token wall against our 240.5; the GEMV alone is 210 ms of that, so even a
PERFECT GEMV leaves ~20 ms and the rest of the gap is the ~30 ms/token of non-GEMV
cost. The binding constraint on this kernel is DRAM bandwidth on a 50 GB working
set streamed once per token, and it is 9% from the roof.

SIDE FINDING, WORTH MORE THAN THIS ROW'S 1%. The ~1.8x bimodal decode legs this
box produces, which the previous Vulkan row discarded as environmental, are
vt_matmul -- the lm_head, NN orientation, ONE call per token -- collapsing by
19.5x: 12.35 ms/token in the fast mode and 242.12 ms/token in the slow one, in the
same two-length diff, while vt_matmul_vec beside it moved 1.2%. At 2.54 GB of
lm_head weights that is 205 GB/s versus 10.5 GB/s. Mean TTFT is unchanged across
the modes (6.21 to 6.55 s over all 28 legs), so it is neither the load nor
prefill. Whatever selects that state is a bigger lever than every remaining
Vulkan kernel optimisation combined.

CORRECTNESS. The load width repartitions K across lanes, so it changes the
answer's low bits exactly as the tactic itself does and is gated on token
exactness, not on an NMSE bound. New case "vt_matmul_vec selects its load width
and row count from the shape" asserts the SPECIALIZATION VALUES the dispatch
built, not the numbers: every value of both axes computes the same dot product to
within this kernel's tier, so a tolerance test cannot see the optimisation at all
and would still pass if the predicate silently stopped selecting it. It pins the
shipped default and the whole degradation ladder (K = 2 mod 4 halves the width,
odd K falls to one element, an f32 activation declines), and carries numeric
checks including the LAST output row, where a wide-load or row-count mistake at
the end of a dispatch leaves every earlier row right. Scratch mutation: reverting
kGemvPackDefault to 0 turns it red (6 of 33 assertions), restoring it green.

GATES, all on GB10 unless noted. test_vulkan_backend 30/30 (2366 assertions) on
GB10 and 30/30 (1823) on llvmpipe -- one case and 37 assertions more than the
previous 29/1786 because of the new case. test_backend_cross_device 11/11 (132).
test_opt_paged_engine with VLLM_CPP_DEVICE=vulkan 6/6 prompts token-exact (96/96
tokens), 0 declines. gen-vulkan-spirv.py --check clean at pinned glslang 16.5.0.
All nine (rows, pack) arms verified numerically on llvmpipe. check-public-doc-
tables and check-env-doc still report the same three and twelve pre-existing
findings they report on 93852c2, none of them in this change.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]